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Abstract 

This paper describes the Entropy Reduction En- 
gine, an architecture for the integration of plan- 
ning, scheduling, and control. The architecture is 
motivated, presented, and analyzed in terms of its 
different components; namely, problem reduction, 
temporal projection, and situated control rule exe- 
cution. Experience with this architecture has mo- 
tivated the recent integration of learning, and this 
paper also describes the learning methods and their 
impact on architecture performance. 

1 Introduction 

This paper describes the Entropy Reduction Engine, an ar- 
chitecture for the integration of planning, scheduling, and 
control. The next section motivates the architecture, and 
the main body of the paper describes the architecture ac- 
cording to various dimensions. This architecture has been 
tested on some simple but representative problems, and un- 
less otherwise noted, all capabilities that are described have 
been implemented. 

2 Architecture Motivation 

At the outset, we would like to indicate what we mean by the 
word “integrated”, as used in the phrase “integrated archi- 
tecture”. To do this, we must first discuss our target problem 
class. 

We are addressing a class of problems that heretofore have 
been largely considered in piecemeal fashion. The problems 
are those that require planning, scheduling, and control. The 
Entropy Reduction Engine (ERE) project is a focus for re- 
search on planning and scheduling in the context of closed- 
loop plan execution. The objective of the project is to cre- 
ate a set of software tools for designing and deploying inte- 
grated planning and scheduling systems that are able to ef- 
fectively control their environments (Bresina & Drummond, 
1990). This objective has two important subgoals: first, we 
are working to integrate planning and scheduling^ second, we 
are studying plan execution as a problem in control 

Traditional AI planning is mainly concerned with the selec- 
tion of actions that are relevant to achieving given goals. Var- 
ious disciplines, principally Operations Research, and more 
recently AI, have been concerned with the scheduling of ac- 
tions; that is, with sequencing a given set of actions in terms 
of metric time and metric resource constraints. Unfortu- 
nately, these two bodies of work remain theoretically and 
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practically disconnected from each other. It is clear that the 
choices made in planning must influence subsequent schedul- 
ing, but it is also true that choices made in scheduling can 
engender further planning activity. The ERE architecture in- 
tegrates planning and scheduling functions so that scheduling 
decisions can give rise to further planning activity. 

Most planning and scheduling work assumes that the job of 
the software system is done when a plan has been gener- 
ated. However, as Dwight D. Eisenhower observed, “Plans 
are nothing, planning is everything”. We agree with this view 
in the sense that the importance of planning does not lie in 
the existence of a single plan, but rather in a system’s ability 
to predictively manage plan execution in light of continuous 
feedback from an environment and to re-plan when failures 
occur. In the ERE project, we formalize plan execution as 
a form of closed-loop control, where a plan describes a de- 
sired behavior, and feedback from the environment is used to 
measure deviations. 

Diversity in the class of problems poses both the difficulties 
and opportunities of architectural integration. We are using 
various problem-solving methods in our architecture, includ- 
ing problem reduction, temporal projection, and rule-based 
execution. We also intend to employ both analytic and in- 
ductive learning methods. The problem-solving methods that 
are being integrated have been selected due to their apparent 
relevance to problem types of concern (planning, scheduling, 
and control). We are pursuing this integration effort with the 
hope of achieving more from the methods’ interaction than 
could be achieved from any single method in isolation. 

3 Architecture Components 

The ERE architecture consists of three major components: 
the Redactor, the Projector, and the Reactor. The Reduc- 
tor synthesizes appropriate problem-solving strategies for a 
given problem; the Projector uses these strategies as search 
control to plan and schedule appropriate actions; and the 
Reactor executes control rules derived from the Projector’s 
plans. Bresina and Drummond (1990) present an overview 
of this architecture; here, we only discuss how ERE can be 
used to build a system for a particular problem. We take a 
programming language view on the architecture and consider 
the different sorts of knowledge that a user must provide in 
order to construct an application system. 

First, a user must provide a causal theory for the domain of 
application, or no behaviors can be produced by the system. 
This theory consists of a description of the control actions 
that can be taken by the system, their preconditions, and 
probability distributions of these actions’ possible effects. In 
a similar fashion, a user can also specify exogenous events 
that are outside of the system’s control. This information is 
used by the Projector to reason about possible system behav- 
iors. To complete the causal theory, the user must provide 
domain constraints that specify those facts which can never 
co-occur; these constraints are used throughout the system to 



By taskability we mean the ability of a system to accept new 
goals at run time. ERE has this ability, provided that there 
is time for the three components to react. The Reactor can 
accommodate a new goal with ease, provided that it has ap- 
propriate SCRs. If no such SCRs exist, then the Projector 
must carry out a search and compile SCRs for the new goal. 
If the current strategy is inappropriate for the new goal, then 
the Reductor must respond by producing a new and appro- 
priate strategy. It is possible to change goals at run time 
because we do not wire a single goal into the system at de- 
sign time. Of course, there is a cost associated with this 
run-time flexibility: in order to produce a new set of SCRs, 
each system component might have to carry out extensive 
computation. 

5.4 Adaptability 

By system adaptability we refer to the introduction of learn- 
ing methods that enable the system to improve itself over 
time. One way to accomplish this is by acquiring and refining 
various sources of knowledge. Currently, there are three types 
of knowledge in the system that are acquired and refined: the 
causal theory, the SCRs, and the problem reduction rules. In 
this section, we consider each in turn. 

First, the causal theory can be refined by detecting and recov- 
ering from failures (Kedar, Bresina, &: Dent, 1991). In par- 
ticular, errors in the causal theory are detected when there is 
a discrepency between what was projected to occur and what 
actually occurred during reaction. These ‘prediction failures’ 
may be caused by one (or more) of the following: missing op- 
erator preconditions, missing operator outcomes, or missing 
domain constraints. We have implemented a technique us- 
ing explanation-based learning (Mostow & Bhatnagar, 1987; 
Gupta, 1987; Chien, 1989; Minton, 1988b) to acquire gen- 
eral preconditions to be added to an operator schema. We 
are currently developing techniques to handle other kinds of 
missing information. These techniques require the addition 
of induction to explanation- based learning. 

Secondly, SCRs can be acquired and refined through caching 
goal-satisfying behaviors synthesized by the Projector, The 
most specific version of this process is a form of knowledge 
compilation. However, SCRs can be formed with varying de- 
grees of generality using a goal regression algorithm (Mitchell, 
et a/., 1986) that we have extended to regress goals with tem- 
poral extent. Compiling general SCRs using goal regression 
is very similar to Soar’s chunking mechanism (Laird & Rosen- 
bloom, 1990) and to Theo’s use of explanation-based learning 
(Mitchell, 1990). 

Thirdly, problem reduction rules can be refined in the face of 
inappropriate problem-solving behavior. Recall that the Re- 
ductor synthesizes an ordered set of subproblems which the 
Projector attempts to satisfy in turn. Whenever a subprob- 
lem is solved, the solution is compiled into SCRs which are 
immediately made available to the Reactor. Thus, if execu- 
tion must begin before a complete solution has been found, 
the Reactor is guided by the SCRs resulting from the current 
solution prefix. However, harmful interactions between sub- 
problems can make it impossible to extend the partial solu- 
tion to satisfy the entire strategy. If this happens the Reactor 
may have to physically backtrack. Hence, it is important that 
the probability of backtracking over a subproblem solution be 
kept low. This probability can be reduced by incrementally 
refining the problem reduction rules, such that the subprob- 
lems they specify are more independently solvable. We are 
currently developing a combination of analytic and inductive 
learning mechanisms to address this issue. 


5.5 Scalability 

Any given problem can be scaled up in a number of ways; 
in general, the major impact on our architecture is an in- 
crease in the size of the projection search space. In order for 
the synthesized SCRs to be of sufficient “quality” within the 
available time, the search guidance supplied by the Reductor 
must also be of sufficient “quality”. The quality of the syn- 
thesized problem solving strategies is dependent (mainly) on 
the expertise encoded in the problem reduction rules. Thus, 
if the expertise appropriate for the scaled-up problem is sup- 
plied (or learned), the Reductor has the potential to effec- 
tively guide the Projector’s search. However, this potential 
can only be realized if there is enough computation time avail- 
able. Scaled-up problems tend to require deeper searches in 
the reduction space and in the projection space; hence, the 
anytime characteristics will degrade. Partially due to this 
degradation, if all planning search must be carried out con- 
current with action, the Reactor’s response time could in- 
crease. The other (potential) reason for increased response 
time is that scaled-up problems tend to require a larger set of 
SCRs to specify the appropriate control advice. We’re cur- 
rently studying system performance in the TileWorld simu- 
lator (Philips & Bresina, 1991), but plan to address the seal- 
ability issue by using ERE on a more realistic NASA domain 
in the future. 

5.6 Reactivity 

Environmental change potentially affects all three ERE com- 
ponents. The Reactor can easily respond to environmental 
change, and in fact, that is exactly its job. On each cycle, 
it checks for applicable SCRs (relevant to the given goal and 
enabled in the current situation) and executes one of the indi- 
cated actions. Clearly, if the sensor values indicate a situation 
for which there is an applicable SCR, then the Reactor can 
accommodate environmental change. Things are more prob- 
lematic when we consider how the Projector can respond to 
environmental change. This problem is essentially that of 
having a planning system recognize when assumptions it has 
made about the environment are invalidated during plan con- 
struction. We do not currently have a solution to this prob- 
lem implemented, but we are considering various dependency 
tracking mechanisms. Lastly, the Reductor must also notice 
environmental change, since it is possible that the currently 
selected strategy is inappropriate in the current situation. 
This is an open research problem. 

5.7 Efficiency 

The maximum response time of the Reactor occurs when it is 
forced to respond (i.e., perform some action) and no available 
SCR is applicable. Since the causal theory specifies all possi- 
ble actions, the Reactor can respond by (randomly) selecting 
one of the enabled actions. Hence, the maximum response 
time equals the time needed to determine that no SCRs are 
applicable plus the time to determine which actions, speci- 
fied in the causal theory, are enabled in the current situation. 
This calculation depends on the match algorithm, as well as 
on the size and, more importantly, on the organization of 
the set of SCRs. The time required to guarantee an “appro- 
priate” response depends on whether the necessary planning 
has been done in advance, and if not, it depends on the any- 
time characteristics of the Reductor and Projector. We have 
not yet addressed the utility problem; that is, the problem 
of SCRs with excessively expensive applicability conditions. 
This problem is especially acute when using generalized SCRs 
(Minton, 1988a). 
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